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1.0  OBJECTIVE 


The  Human  Systems  Center's  Study  and  Analysis  Division  (HSC/XRS)  would  like  to  develop  a  metric  to 
measure  our  customers'  satisfaction.  Customer  feedback  is  the  cornerstone  for  improving  our  work 
performance.  The  vehicle  used  to  measure  customer  satisfaction  should  be  easy  to  administer  as  well  as 
easy  to  understand.  It  should  be  realistic  and  be  used  as  a  tool  for  continuous  improvement  of  services. 


2.0  BACKGROUND 


2. 1  Our  Way  of  Life 

Total  Quality  Management  (TQM)  is  a  "way  of  life"  for  the  Air  Force.  Striving  to  do  a  better  job  while 
meeting  our  customers'  needs  is  the  critical  tenet  behind  TQM.  To  ensure  that  we  are  accomplishing  this 
tenet,  one  must  query  the  customer.  One  key  way  is  through  the  implementation  of  a  metric  and  seeing 
how  we  measure  up  to  it.  Especially  in  these  times  of  tight  budgets,  it  is  critical  that  we  get  feedback  from 
users  to  ensure  that  they  are  getting  the  most  from  their  limited  resources.  But,  what  should  be  measured 
and  how  should  we  go  about  measuring  it?  Our  office  has  been  struggling  with  this  concept  for  over  two 
years. 

2.2  What  Our  Office  Does 

HSC/XRS  deals  with  long  range  developmental  planning,  pre-acquisition  studies,  modeling,  and  cost 
analyses.  The  majority  of  our  work  are  paper  studies  per  the  request  of  users,  such  as  MAJCOMs.  Topics 
range  the  full  gamut  of  early  acquisition  studies  in  human-centered  areas.  Some  studies  are  of  short 
duration,  lasting  less  than  a  year,  with  most  averaging  about  two  years.  Studies  do  not  regularly  lead  to 
HSC  follow-ons  due  to  the  nature  of  the  topics;  usually  action  taken  as  a  result  of  a  study  will  not  fall 
under  the  HSC  mission. 

2.3  Developing  The  Metric 

To  develop  a  viable  metric  for  HSC/XRS,  the  author  decided  to  first  query  co-workers  throughout  XR  as 
to  what  should  go  into  the  metric  and  how  it  should  be  administered.  A  questionnaire  was  the  approach, 
the  author  chose  to  use.  A  literature  search,  delving  into  what  our  industry  counterparts  are  doing,  also 
was  used  to  help  form  our  metric  and  the  means  to  measure  it.  Discussions  with  the  HQ  HSC  TQM  focal 
point,  Ms  Virginia  Moody,  provided  valuable  insight.  Also  a  three  month  collection  of  customer 
satisfaction  questionnaires  received  from  both  industry  and  DoD  was  compiled  for  analysis  and  helped 
form  the  foundation  of  this  effort. 


3.0  METHOD 

3. 1  The  Questionnaire 

Attachment  1  contains  the  questionnaire  developed  by  the  author  to  query  individuals  on  what  they  think 
should  be  in  the  XRS  Customer  Satisfaction  Metric.  Her  questionnaire  was  tested  for  continuity  by  Maj 
Sipes,  a  PhD  Psychologist,  working  in  HSC/XR.  She  also  discussed  it  with  Survey  Control,  Randolph 
AFB;  a  survey  control  number  was  not  required  per  their  discretion. 
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3.2  Test  Population 

The  questionnaire  was  sent  out  to  26  individuals  throughout  HSC/XR,  both  military  and  civilians.  Ten 
work  days  were  given  each  recipient  to  complete  the  survey.  The  questionnaire  was  also  faxed  to  a  focal 
point  at  HQ  AMC.  This  key  point  of  contact  of  the  author's  at  HQ  AMC  answered  the  questionnaire, 
providing  the  user's  perspective. 


4.0  RESULTS 

Results  from  XR  and  the  customer,  HQ  AMC,  are  broken  out  and  described  below. 

4.1  XR  Results 

20  completed  surveys  were  returned.  This  is  a  return  rate  of  77%.  The  following  responses  were 
received: 

4.1.1  Question  1:  How  often  should  customer  satisfaction  be  measured  via  questionnaires, 


etc.? 

Response  %  of  Respondents 

End  of  effort  only .  24.0% 

Quarterly .  9.5% 

At  key  milestones .  57.0% 

No  item  checked  -  only  comments .  9.5% 


One  person  marked  two  entries  ("end  of  effort  only'V’at  key  milestones");  therefore  above  statistics  are 
based  on  21  responses  rather  than  20.  60%  of  the  people  marking  "end  of  effort  only"  also  had  a 
comment,  such  as:  "or  end  of  year";  "or  every  other  year  which  ever  comes  first".  One  person  who 
marked  "at  key  milestones"  entered  a  comment  on  their  questionnaire:  "would  suggest  this  for  long  studies 
-  for  shorter  studies,  possibly  mid  and  end  of  study".  Of  the  two  respondents  with  only  comments,  the 
following  were  their  responses:  "should  be  no  longer  than  1  year  nor  more  frequent  than  quarterly"; 
"depends  on  goals.. .all  are  applicable". 

4.1.2  Question  2:  Give  three  key  issues  that  should  be  addressed. 


Response  %  of  Respondents 

Cost .  6.9% 

Performance  . . 6.9% 

Schedule/Timeliness .  20.3% 

Responsive  to/satisfaction  of  needs  and  expectations .  19.6% 

Are  any  changes  required . 1.7% 

Courtesy/bed  side  manner/helpfulness . . .  5.1% 

Subject  matter  experts .  1.7% 

Individual  initiative .  1.7% 

Do  we  do  follow-ups  without  customer  request .  1.7% 

Quality . 6.9% 

Is  the  product  useful/valuable .  6.9% 

Are  all  stakeholders  involved  in  study .  1.7% 
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Does  product  present  salient  points . . .  1.7% 

Incorporation  of  customer  info .  3.5% 

Are  documented  requirements  fully  addressed .  1.7% 

Customer  kept  "in-the-loop" .  1.7% 

Constructive  criticisms/suggestions  for  improvement  .  3.5% 

Willingness  to  work  concerns .  1.7% 

Level  of  concern . 1.7% 

Value  of  work  by  XRS  versus  competitor .  1.7% 

Dissatisfaction/problems .  1.7% 


One  person  gave  four  responses  and  one  person  did  not  give  any;  therefore,  above  statistics  are  based  on 
58  responses  rather  than  60. 

4.1.3  Question  3:  Rank  order  the  following  in  order  of  importance  (1  for  most  importance-7 
for  least)  of  what  should  be  included  in  any  survey  regarding  customer  satisfaction. 


%  of  Respondents 


Response  Average  Value 

Listing  as  #1 

Ranee 

XR's  management  of  effort . 

4.5 . 

.  5% . 

. 1-7 

Customer's  needs  addressed . 

1.5 . 

.  75%  .... 

.  1-4 

Customer's  request  for  follow-on  work . 

4.9 . 

.  0 . 

. 2  -N/A 

Willingness  of  customer  to  spend  own  money  . 

5.6 . 

.  0 . 

. 3  -N/A 

Timeliness  of  XR . 

3.4 . 

.  0 . 

. 2-6 

Use  of  results  by  customer . 

4.3 . 

.  10%.... 

.  1  -N/A 

A,  B  or  C  of  item  #2 . 

- 

.  10% .... 

.  1-7 

Not  applicable  (N/A)  was  counted  as  a  7.  14  people  gave  responses,  some  more  than  one  (total  of  18 
responses),  to  the  last  item,  "A,  B  or  C  of  item  #2";  therefore  the  average  value  was  not  listed  to  avoid 
confusion.  But,  of  the  people  that  did  answer  this  question,  their  average  value  was  3.4.  Two  people  gave 
"A,  B  or  C  of  item  #2"  a  value  of  1;  their  responses  were:  "Quality"  and  .  , 

"Schedule/Cost/Performance/Responsiveness".  Six  people  gave  "A,  B  or  C  of  item  #2"  a  value  of  2;  their 
responses  were:  "Quality"  (x3),  "Incorporation  of  customer  info",  "Helpfulness",  and  "Cost/Performance". 
For  the  remaining  six  people,  all  but  one  had  their  "A,  B,  or  C  of  item  #2"  already  listed  in  this  question. 

4.1.4  Question  #4:  What  rank  or  position  should  you  send  the  survey  to?  Does  it  make  a 
difference,  as  long  as  you  send  it  to  the  primary  customer?  Why? 

20%  of  the  respondents  said  that  the  survey  should  go  to  a  high  ranking  officer  (Col  or  above).  60%  of 
the  respondents  said  that  the  person  with  intimate  knowledge/the  actual  user  should  get  the  survey;  one 
quarter  of  this  sub-group  said  that  the  supervisor  should  also  get  the  survey.  The  remainder  of  the 
comments  were:  "to  the  two  letter";  "depends  on  the  target  population",  "tasking  office". 

Regarding  whether  or  not  it  makes  a  difference  to  whom  you  send  the  survey,  the  following  comments 
were  received:  "people  are  people,  rank  or  no  rank";  "high  enough  for  the  big  picture,  low  enough  for 
knowledgeable  input";  "you  do  not  want  a  lieutenant  answering  the  survey,  they  do  not  control  funding"; 
and  "actual  user  regardless  of  rank".  Most  respondents  did  not  answer  this  part  of  question  #4. 
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4.1.5  Question  #5:  Are  customer  surveys  a  valid  way  of  determining  what  the  customer 
thinks?  Do  you  think  they  are  fire-walled? 

60%  of  the  respondents  claimed  that  surveys  are  valid.  One  person  who  claimed  that  surveys  were  valid 
also  stated  that  you  need  to,  "follow-up  with  a  teleconference".  20%  described  them  as  so-so.  20%  said 
that  surveys  are  not  valid.  Comments  received  were,  "theory  yes,  reality  no";  "we  are  over-doing  it";  "but  if 
really  unhappy,  they  will  tell  you";  "surveys  tell  you  more  than  person-to-person";  "limited  use,  just  a 
snapshot  of  what  they  feel  at  the  moment";  and  "I  think  many  people  blow  them  off  (but  not  me  on  this 
survey)  and  even  the  conscientious  often  don't  have  the  time  or  skills  to  accurately  communicate  the  whole 
picture". 

Not  everyone  answered  the  second  part  of  this  question.  10%  answered  that  they  did  not  know  what  the 
word  "fire-walled"  meant  (these  two  people  were  civilians).  25%  said  that  they  were  fire-walled  often 
while  20%  said  they  were  not.  A  comment  received  was:  "if  anonymous  or  voluntary  name,  fire-walling 
may  not  occur". 

4.1.6  Question  #6:  What  else  besides  survey  forms  could  we  use  to  measure  customer 
satisfaction?  Which  do  you  prefer? 

45%  said  that  you  should  ask  them  personally:  "eye-ball  to  eye-ball  discussion  with  point  'how  did  we  do' 
and  'how  can  we  improve'  discussions";  "ask  if  they  are  using  survey  findings";  and  "inquire  if  what  we  are 
doing  is  to  their  liking".  10%  said  that  we  should  strive  towards  a  metric  that  is  quantitative,  "did  they 
request  more  work  from  us";  "did  they  change  what  they  were  doing  or  going  to  do";  "response  time  to 
requests";  "number  of  contacts/problems  solved".  Other  comments  were:  "survey  forms  are  the  best";  "I 
wish  I  knew  something  to  suggest,  there  are  already  too  many  customer  surveys  going  around";  and  "good 
luck". 

4. 1 .7  Question  #7:  Comments: 

The  following  comments  were  received:  "what  does  industry  use?";  "one  bad  comment  doesn't  mean  to 
make  drastic  changes"; "  survey  can  provide  a  feedback  loop";  "what  empowerment  are  you  given  to  make 
changes  from  your  survey  results  --  we  can  not  change  regs  and  systems";  "perhaps  developing  a  verbal 
survey  or  checklist  —  verbally  discussing  customer  satisfaction  with  the  customer  at  key  points";  "keep 
survey  short";  "nice  effort";  and  "good  luck". 

4.2  User  Response 

The  user  felt  that  customer  satisfaction  should  be  measured  both  at  key  milestones  and  at  the  end  of  the 
effort.  Regarding  key  issues  that  should  be  addressed,  the  user  weighed  all  six  of  the  following  as  equally 
important:  "customer's  needs  met";  "communication";  "timeliness";  "periodic  updates";  "scheduling";  and 
"program  management".  The  user  also  felt  that  rank  did  not  have  anything  to  do  with  this  type  of 
questionnaire:  "when  we've  received  them  in  the  past  at  senior  officer  level,  they  just  send  it  back  to  the 
action  officer  because  we  deal  with  the  provider  more  than  them"’.  The  user  felt  that  customer  satisfaction 
surveys  are  valid,  but  also  recommended  that  a  follow-up  a  year  later  may  help.  In  response  to  Question 
#7,  the  user  made  reference  to  the  author's  primary  program,  Passenger  Smoke  and  Fume  Protective 
Device  (PSFPD):  "  You've  done  a  great  job  on  PSFPD . Thanks!" 
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5.0  DISCUSSION 


5.1  XRS  Guidelines 

After  compiling  the  raw  data,  the  author  handed  it  to  her  co-workers  in  XRS  for  their  comments.  A  set  of 
guidelines  were  generated.  XRS  personnel  unanimously  agreed  on  the  following  guidelines: 

5.1.1  Use  Questionnaire  with  Follow-up.  Although  not  perfect,  a  properly  worded  questionnaire 
with  frequent  up-dating  can  allow  the  customer  to  provide  adequate  feedback.  Personalizing  the 
questionnaire  with  a  phone  call  within  two  weeks  of  sending  the  questionnaire,  will  hopefully  insure  a  good 
response  rate  plus  generate  additional  quality  feedback. 

5.1.2  Administer  at  Key  Milestones.  Not  all  key  milestones  warrant  a  questionnaire  for  feedback. 
Also,  end  of  effort  will  be  counted  as  a  key  milestone  and  should  warrant  a  mandatory  questionnaire. 

5.1.3  Keep  Questionnaire  Short.  Questionnaire  should  not  exceed  one  page  in  length. 

5.1.4  Ensure  Compatibility  with  Front  Office.  Apparently  other  organizations  on  Brooks  are  using 
a  1-6  rating  scale  on  questionnaires.  This  currently  used  format  allows  for  no  middle  ground,  as  would  an 
odd  numbered  scale. 

5.1.5  Send  to  Highest  Person  with  Working  Knowledge  of  Project. 

5.1.6  If  Customer  is  Upset,  Rectify  Problem  and  Re-send  Survey.  This  should  take  place  as 
quickly  as  possible;  if  problem  is  not  solvable  within  the  month,  contact  should  be  made  with  customer 
advising  them  of  the  steps  taken  to  alleviate  the  problem. 

5.2  Developing  the  Questionnaire 

Based  on  the  raw  data,  a  questionnaire  was  prepared  to  measure  customer  satisfaction.  It  was  limited  to 
one  page,  with  only  4  questions  plus  one  section  for  comments  (see  Attachment  2).  The  title  of  "Customer 
Feedback"  was  chosen  to  stimulate  a  dialog  between  the  customer  and  XRS,  rather  than  just  rating  their 
satisfaction.  The  first  three  questions  combine  the  salient  points  raised  by  in  the  raw  data.  These  yes/no 
questions  also  leave  room  for  comments.  The  last  question  used  the  Brooks  1-6  rating  scale.  It  was  felt 
that  this  could  be  used  as  the  over-all  score  if  we  were  tasked  to  report  the  metric.  By  asking  for 
"constructive  feedback",  rather  than  just  for  comments,  we  can  pin  the  customer  down  to  providing  useful, 
valuable  information  that  can  be  used  to  improve  our  services. 


6.0  CONCLUSION 


6. 1  We  will  implement  a  survey  to  measure  customer  satisfaction.  We  will  join  the  millions  of  others 
throughout  industry  and  the  government  trying  to  improve  their  services  through  user  feedback.  But  we 
realize  that  this  is  a  living  document.  It  must  be  changed  if  we  are  not  getting  the  type  of  responses  that 
are  useful.  We  must  also  keep  in  mind  that  we  need  to  up-date  the  survey  if  our  mission  changes,  user 
feedback  dictates  so,  etc. 

6.2  This  effort  will  help  our  office  meet  one  of  seven  published  Air  Force  Materiel  Command  Total 
Quality  Management  objectives: 
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6.2.2  "...develop  a  series  of  metrics  by  which  we  will  be  able  to  assess  our  progress."1 


6.3  Comments  on  this  study/survey  are  appreciated. 

7.0  REFERENCES 


Gerson,  Richard  F.  (1993).  Measuring  Customer  Satisfaction.  Published  by  Crisp  Publications,  Inc. 

Keiser,  Allyn  (1993).  "Keeping  the  Customer  Satisfied  Begins  with  Asking  Questions".  Bank  Management 
v69,  pp  48-51 

Pfau,  Bruce  (1993).  "Surveys  Used  as  Weapons  in  Fight  for  Survival".  National  Underwriter  v97,  ppl  1-12 

US  Air  Force  Materiel  Command  pamphlet  (1992),  An  Introduction  to  Total  Quality  Management  in  Air 
Force  Materiel  Command 


'US  Air  Force  Materiel  Command  pamphlet  (1992).  An  Introduction  to  Total  Quality  Management  in  Air  Force  Materiel 
Command  pp5 


7 


ATTACHMENT  1: 

Questionnaire  For  Co-workers  and  Customers 
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DEVELOPMENT  OF  A  CUSTOMER  SATISFACTION  METRIC 

Questionnaire  authored  by:  Capt  Eileen  Ancman 
HSC/XRS,  Brooks  AFB  TX 
DSN240-4456 

PLEASE  RETURN  BY: _ 


Name  w/rank: _ 

Organization/phone:. 


The  Studies  and  Analysis  Division,  HSC/XRS,  of  Brooks  AFB  desires  to  develop  a  metric  to 
measure  customer  satisfaction.  Your  answers  to  the  following  questions  will  help  us  in  this 
endeavor  and  are  greatly  appreciated. 


1 .  How  often  should  customer  satisfaction  be  measured  via  questionnaires,  etc.? 

end  of  effort  only _ 

quarterly _ 

at  key  milestones _ 

other _ 


2.  Give  three  key  issues  that  should  be  addressed: 


a. 


b. 


c. 


3.  Rank  order  the  following  in  order  of  importance  (1  for  most  important  --  7  for  least)  of  what 
should  be  included  in  any  survey  regarding  customer  satisfaction  : 

_  XR's  management  of  effort  _  timeliness  of  XR 

_ _  customer's  needs  addressed  _  use  of  results  by  customer 

_  customer 's  request  for  follow-on  work 

_ _  willingness  of  customer  to  spend  own  money  to  attend 

meetings,  etc. 

_  A,  B,  or  C  of  item  #2  (circle  which  applies,  if  any) 


9 


4.  What  rank  or  position  should  you  send  the  survey  to?  Does  it  make  a  difference,  as  long 
as  you  send  it  to  the  primary  customer?  Why? 


5.  Are  customer  surveys  a  valid  way  of  determining  what  the  customer  thinks? 
Do  you  think  they  are  fire-walled? 


6.  What  else  besides  survey  forms  could  we  use  to  measure  customer  satisfaction?  Which  do 
you  prefer? 


7.  Comments: 
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ATTACHMENT  2: 

Customer  Satisfaction  Questionnaire 
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CUSTOMER  FEEDBACK 


Date: _ 

HSC/XRS  Effort: _ 

HSC/XRS  Program  Manager: _  Phone: _ 

Customer's  Name: _  Phone: _ 

Customer's  Organization/Address: _ ; _ 

Major  Milestone: _ 

1 .  Were  your  needs/expectations  met  at  this  milestone?  (YES  /  NO)  If  not,  please  explain. 


2.  Did  we  keep  you  in  the  loop,  utilizing  a  courteous  demeanor?  (YES  /  NO)  If  not,  please  explain. 


3.  Was  the  information  provided  both  timely  and  useful?  (YES  /  NO)  If  not,  please  explain. 


4.  Please  rate  your  overall  satisfaction  level. 

unsatisfied  1  2  3  4  5  6  satisfied 

5.  Please  use  the  remainder  of  this  form  to  provide  constructive  feedback,  allowing  us  to  serve  you  better: 
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